Nvfp4 four over six grouped experts - #9
Draft
wolfcomos wants to merge 6 commits into
Draft
Conversation
Extends the four-over-six integration to the miles NVFP4 RL recipe shape
(routed experts only, row-scaled activations, 1x16 weights, backward
overrides):
- NVFP4FourOverSixLinear / NVFP4FourOverSixLinearConverter gain
backward_override ('quantized' | 'high_precision' | 'dequantized',
mirroring TransformerEngine's NVTE_BACKWARD_OVERRIDE; None keeps the
previous defaults) and weight_block ('1x16' mirrors
NVTE_NVFP4_DISABLE_2D_QUANTIZATION=1). Knob combinations are validated
at config time.
- NVFP4FourOverSixGroupedExpertsConverter: class-factory _grouped_mm
override calling torchao four_over_six_grouped_mm (like the MXFP8
grouped-experts converter, with fqns include-list filtering). Four-
over-six needs no RHT sign vector and no stochastic-rounding seed, so
the stateless hook carries everything. Token dispatchers swap to
padded variants (pad_multiple=128).
- nvfp4_bf16_first_last_fqns: the miles --first-last-layers-bf16 analog.
- deepseek_v3_debugmodel_nvfp4_four_over_six[_dequantized] recipes: the
miles base point (row-scaled + MSE + bound 256 + 1x16 weights +
high_precision backward, experts-only allow-list) and the GLM-5.2
analog (dequantized backward, first/last decoder layer bf16).
- rl_grpo_qwen3_30b_a3b_varlen_nvfp4_four_over_six: the RL wiring. The
trainer and vLLM generator share one model_spec, so both actors run
the identical four-over-six forward -- the train/inference-consistency
point of the miles recipe, with bf16 master weights syncing as usual.
Unit tests: 35 passed (TE 2.19 devel container), covering converter
targeting, knob plumbing, first/last-bf16 windows, and config-time
validation. GPU smoke on GB200 verifies the hook's (E, K, N) -> (E, N, K)
weight orientation, ragged-group padding, and both grads end to end.
Depends on the TorchAO branch nvfp4-four-over-six-rowwise
(wolfcomos/ao#7, through commit c47af4a7e).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TorchAOTokenDispatcher swapped in by the grouped-experts converter already 128-aligns every expert group, but it over-allocates the activation buffer past offs[-1], while torchao's four_over_six_grouped_mm requires offs[-1] == A.shape[0]. Passing the buffer through with pad_token_groups_for_grouped_mm=True violates that contract, and simply disabling padding would let the unwritten tail rows feed the per-group amaxes and silently poison the last expert's quantization in per-tensor mode. Slice the logical rows before the op, skip the op's own padding (the dispatcher's alignment already satisfies it), and zero-extend the output so downstream shapes match; pad routes zero gradients to the tail and the unpermute never reads those rows.
The quantized grouped GEMM reads expert group offsets on the host and the row-scaled mode loops dense GEMMs per group, both of which CUDA-graph capture forbids (cudaErrorStreamCaptureUnsupported in the debugmodel's captured fwd+bwd). miles runs its quantized recipes with CUDA graphs off for the same class of reason.
The row-scaled grouped forward host-reads the group offsets and loops dense GEMMs per group; under torch.compile fullgraph (nonstrict_trace + capture_scalar_outputs) the offsets become unbacked SymInts and the loop bounds guard on data-dependent expressions with no graph-break escape. Fail at config time instead of at trace time.
has_quantization enumerated only the Float8/MXFP8/NVFP4Linear configs and the float8/mxfp8 grouped-experts caches, so a model quantized by the four-over-six converters reported has_quantization=False and MFU was computed against the bf16 peak.
Covers the config-time ValueError for model compile + row-scaled grouped four-over-six, and has_quantization returning True for models converted by either four-over-six converter (and False for the stock debugmodel).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.